Estimating Relevance and Semantic Compatibility for IE Pattern Discovery in Large Text Corpora

Pattern-based approaches for Information Extraction (IE) typically apply a pattern learner to a set of domain-specific training documents to generate extraction patterns for the IE system. This restricts the coverage of the system primarily to the expressions and language constructs that appear within the limited training data. Our research looks to the vast quantities of readily available text in resources like the Gigaword Corpus to expand the coverage of existing pattern-based IE systems. The learning strategy exploits two inherent characteristics of extraction patterns – relevance and semantic compatibility – for learning new patterns from unannotated sources. We implement a statistical correlation approach for estimating pattern relevance, and perform semantic compatibility ranking of patterns using two separate approaches – one based on vector similarity, and another based on the semantic classes of extractions. We demonstrate an overall increase in the coverage of an existing IE system using this two-phase strategy for pattern learning.